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Appendix A. Timeline of low-performing school identification, monitoring, and exit decisions 
by the Massachusetts Department of Elementary and Secondary Education 


The Massachusetts Department of Elementary and Secondary Education timeline for low-performing school 
identification, monitoring, and exit decisions is presented in table A1. 


Table A1. Turnaround practice timelines 


Year 1—Identification and baseline of low-performing school 


Fall Low-performing school is identified by the Massachusetts Department of Elementary and Secondary 
Education (DESE) based on the school’s prior-year performance data. (For example, schools identified in 
2015/16 as low performing are identified based on student assessment results from the 2014/15 school 
year.) 
Low-performing school receives an annual monitoring visit. 
Low-performing school receives a report with Turnaround Practices and Indicators (TP&l) ratings. 
The district and the DESE support provider meet with the school leader to discuss the report findings and 
strategies for continuous improvement. 


Winter and Low-performing school submits a turnaround plan based on a needs assessment (including data from the 
spring monitoring visit). 
Spring DESE reviews and approves turnaround plan. 


Year 2—Implementation of turnaround plan and continuous improvement 
Fall Low-performing school implements turnaround plan. 


Fall and winter Low-performing school receives a monitoring visit. 
Low-performing school receives a report on the TP&l ratings. 


The district and the DESE support provider meet with the school leader to discuss the report findings and 
strategies for continuous improvement. 


Spring Low-performing school revises turnaround plan based on needs identified in monitoring visit reports. 
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Year 3—Eligibility for exit decision or continuation in accountability status 


Fall Low-performing school implements turnaround plan. 


Low-performing school receives a report on the TP&lI ratings. 
The district and the DESE support provider meet with the school leader to discuss the report findings and 


strategies for continuous improvement. 
Year 4+—Exit, continuation, or receivership decision 
Exit, continuation, or receivership decision for low-performing school is based on schoolwide student 
outcomes and growth based on student assessment results: 


Low-performing school is eligible to exit underperforming status if they are on track to meet stated 


three-year goals in school performance. 
Low-performing school that has made progress but has not met all benchmarks may be kept in 
turnaround status for another school year. 

Low-performing school that continues to decline in performance during the first three years in 
turnaround status can be placed under control of an external receiver. 


Fall and winter 


Source: Massachusetts Department of Elementary and Secondary Education. 


Appendix B. Indicators and ratings continuum for turnaround practice area implementation 


The Massachusetts Department of Elementary and Secondary Education (DESE) developed criteria for rating levels 
for each indicator in the Turnaround Practices and Indicators (TP&l) rubric. As part of the monitoring process for 
low-performing schools, each school receives a TP&I rating based on an analysis of quantitative and qualitative 
data collected annually from low-performing schools. These data are derived from observations of classroom 
instructional practices during monitoring visits; staff survey responses; and interviews and focus groups with 
school leaders, teachers, students, and other key stakeholders. The TP&I ratings are based on a four-point scale 
for each indicator: limited evidence (the lowest rating, 0), developing (second lowest, 1), providing (second 
highest, 2), and sustaining (highest, 3). 


Turnaround Practice Area 1. Leadership, Shared Responsibility, and Professional Collaboration 


The school has established a community of practice through leadership, shared responsibility for all students, and 
professional collaboration (table B1). 


Table B1. Turnaround Practice Area 1. Leadership, Shared Responsibility, and Professional Collaboration 


Turnaround Practices and Indicators (TP&l) rating 


areltersixelg Limited evidence Developing Providing Sustaining 
1.1 Use of Autonomy School leaders have School leaders have School leaders have the School leaders use their 
little to no autonomy to some autonomy to autonomy to make autonomy and authority 
make decisions about make decisions about decisions about key (regarding, for example, 
key elements of the key elements of the elements of the school staffing or school 
school, such as staffing | school (such as staffing day (suchas staffing or | schedule) to focus on 
or length of the school or school schedule) but school schedule) and implementing their 
day. have not yet used this have begun to use this — turnaround plan or 
autonomy or are autonomy to make other efforts to improve 
uncertain about how changes in the school. the quality of teaching 
best to use it. and learning at the 
school. 
1.2 High There is little to no School leaders School leaders School leaders 
Expectations and evidence that the school understand the understand the understand the 
Positive Regard __ prioritizes high importance of high importance of high importance of high 
expectations and expectations and expectations and expectations and 
positive regard among positive regard among positive regard among positive regard among 
leadership, staff, and leadership, staff, and leadership, staff, and leadership, staff, and 
students. students but do not students and implement students and implement 
implement strategies or strategies or activitiesto strategies or activities to 
activities to ensure that ensure that these ensure that these 
these elements are in elements are in place. elements are in place. A 
place. majority of staff believe 


leadership, staff, and 
students have high 
expectations and 
demonstrate positive 
regard. 
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1.3 


1.4 


Vision/Theory 
of Action and 
Buy-In 


Monitoring 
Implementation 
and School 
Progress 
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School leaders have a 
loosely defined theory 
of action or vision along 
with established goals 
and interim benchmarks 
to guide dramatic 
school improvement, 
but the goals and 
benchmarks are not 
used to inform the 
school’s work. There is 
little to no sense of 
urgency or collective 
responsibility for 
realizing school 
improvement. 


School leaders rarely 
prioritize improvement 
initiatives for 
implementation; 
processes or protocols 
are not in place for 
systemic 
implementation. 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


School leaders have a 
defined theory of action 
or vision along with 
established goals, and 
interim benchmarks 
have been 
communicated to some 
staff. A common sense 
of urgency and shared 
ownership for the 
success of all students 
exists among some staff 
and leaders, but not all 
staff members share 
this responsibility. 


School leaders prioritize 
improvement initiatives 
for implementation; 
however, processes and 
protocols for systemic 
implementation are 
emerging or not well 
defined. 


aaeyareliatss 


School leaders have a 
defined and 
communicated theory 
of action or vision along 
with established goals 
and interim benchmarks 
to drive priorities 
related to turnaround 
efforts, and these goals 
and benchmarks are 
understood and 
implemented 
consistently by most 
staff. A common sense 
of urgency and purpose 
for improvement is 
evident among a 
majority of staff, but 
ownership and 
responsibility for 
success of all students 
may still be centralized 
at the principal or 
leadership team level. 


School leaders prioritize 
improvement initiatives; 
processes and protocols 
for systemic 
implementation are well 
defined. A majority of 
staff members are 
aware of the priorities, 
and some monitoring of 
these initiatives takes 
place. 


Sustaining 


School leaders and most 
staff members 
understand the theory 
of action or vision 
driving the priorities 
related to turnaround 
efforts, are familiar with 
the goals and interim 
benchmarks used to 
consistently monitor 
progress (for example, 
at least once a month), 
and identify and 
prioritize the next level 
of work. Acommon 
sense of urgency and 
ownership for the 
success of all students is 
shared among most 
staff, as demonstrated 
through staff discourse 
and actions. 


School leaders are 
actively engaged in 
monitoring 
implementation of 
turnaround efforts, use 
this information to 
prioritize initiatives and 
strategies, communicate 
progress and challenges 
and seek input from 
staff, and continuously 
and systematically 
monitor progress. 
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1.5? 


1.6 


Trusting 
Relationships 


Time Use for 
Professional 
Development 
and 
Collaboration 
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Limited evidence 


Relationships between 
teachers and 
instructional supports 
(such as coaches) are 
not guided by trust; 
teachers feel coaching 
and instructional 
support is judgmental, 
and evidence of 
collaboration among 
staff is limited. 


The schedule includes 
little or no time for 
professional 
development or 
collaboration between 
teachers. 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


Some relationships 
between teachers and 
instructional supports 
(such as coaches) are 
guided by trust, and 
some teachers feel 
instructional support is 
nonjudgmental, but this 
is inconsistent 
throughout the school. 
Some groups of 
teachers may 
collaborate with 
colleagues to share 
strategies, such as 
developing standards- 
based units, examining 
student work, analyzing 
student performance, 
and planning 
appropriate 
interventions. However, 
this is not consistent 
among all staff. 


The schedule does not 
include adequate time 
for professional 
development 
opportunities, 
collaboration time for 
teachers is limited, or 
the available time is not 
used effectively to 
improve teaching and 
learning. 


aaey are liars 


Most relationships 
between teachers and 
instructional supports 
(such as coaches) are 
guided by trust, and 
most teachers feel that 
instructional support is 
nonjudgmental. There is 
evidence that most staff 
at least occasionally use 
collegial relationships to 
share strategies in such 
work as developing 
standards-based units, 
examining student 
work, analyzing student 
performance, and 
planning appropriate 
interventions. 


The schedule includes 
adequate time for 
professional 
development 
opportunities and 
collaboration for most 
teachers. Time is 
generally used well to 
improve teaching and 
learning. 


Sustaining 


Most staff members 
share a relational, trust- 
focused culture with 
each other and their 
instructional supports 
(such as coaches) that is 
solution oriented and 
focused on 
improvement, as 
exemplified by frequent 
collaboration in 
developing standards- 
based units, examining 
student work, analyzing 
student performance, 
and planning 
appropriate 
interventions. Educators 
regularly share their 
strengths and struggles, 
in the spirit of helping 
each other continually 
improve their practice. 


The schedule includes 
adequate time for 
professional 
development 
opportunities and 
collaboration for most 
teachers. There is a 
process in place for 
evaluating the schedule 
using collected data to 
maximize opportunities 
for teacher professional 
development and 
ensure that it helps all 
educators continually 
improve their practice 
(for example, through 
targeted coaching and 
peer observations) and 
collaboration time. 
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Tarelferheele Limited evidence 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


Providing 


Sustaining 


Structures and 
opportunities for 
fostering staff input into 
school decisions and 
initiatives are informal, 
are not well defined, or 
do not exist. 


1.7. =Communication 


with Staff 


There is little to no 
evidence that school 
leadership prioritizes 
building staff capacity to 
sustain improvement 
efforts. 


1.8° Sustainability 


Formal structures and 
opportunities for 
fostering staff input into 
school decisions and 
initiatives are defined 
but may not be used to 
effectively build 
relationships and two- 
way communication 
across staff and school 
teams. 


School leadership is 
aware of the 
importance of planning 
for sustainability. 
However, there is little 
to no evidence that 
improvement efforts 
will be sustained over 
time or under new 
leadership. 


Formal structures and 
opportunities for 
fostering staff input into 
school decisions and 
initiatives are in place 
and are used effectively 
to build relationships 
and two-way 
communication across 
staff and school teams. 
However, there are 
some barriers to 
communication 
between administrators 
and staff. 


School leadership 
implements specific 
strategies (such as 
succession plan, 
distributed leadership, 
and new funding 
streams) to ensure that 
improvement efforts 
will be sustained over 
time or under new 
leadership. 


Formal structures are in 
place to build effective 
staff relationships 
balanced with 
transparency and open, 
two-way 
communication across 
staff and school teams 
and between 
administrators and staff. 


School leadership 
implements strategies 
(such as succession 
plan, distributed 
leadership, and new 
funding streams) to 
ensure that 
improvement efforts 
will be sustained over 
time or under new 
leadership. Majority of 
staff believe and can 
describe specific 
strategies that will 
enable the school to 
continue to improve, 
even with changes in 
staff or school 
leadership. 


a. Instructional Leadership and Improvement (formerly indicator 1.5 in the 2014/15 monitoring site visit report) was removed, and its content was 
incorporated into indicator 2.4 in the 2015/16 monitoring site visit report. Trusting Relationships (formerly Indicator 4.5 in the 2014/15 monitoring site visit report) 


is now reflected in indicator 1.5. 


b. Sustainability (indicator 1.8) was a new indicator in the 2015/16 monitoring site visit report. 
Source: Massachusetts Department of Elementary and Secondary Education. 
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Turnaround Practice Area 2. Intentional Practices for Improving Instruction 


The school employs intentional practices for improving teacher-specific and student-responsive instruction 


(table B2). 


Table B2. Turnaround Practice Area 2. Intentional Practices for Improving Instruction 


axel {erheela MTaalincyem=y"alellare) 


Turnaround Practices and Indicators (TP&I) rating 


1BY=\=1(0) 0) faye 


agente lials4 


Sustaining 


2.1 Instructional Expectations for 
Expectations teachers’ classroom 
practices are not 
articulated by school 
leaders. 
2.2 Instructional Existing instructional 


schedules lack 
consistency or do not 
include uninterrupted 
blocks of schoolwide 
learning time for 
students. 


Schedule 
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Expectations for 
teachers’ classroom 
practices are 
communicated, but the 
expectations may not be 
specific, are not 
understood by most 
staff, and might not be 
actively monitored by 
school leaders. 


Existing instructional 
schedules include 
uninterrupted blocks of 
schoolwide learning 
time. However, 
instructional support 
staff are not coordinated 
and aligned across grade 
levels and content areas 
to provide students with 
differentiated access to 
high-quality core 
instruction. 


Specific or precise 
expectations for 
teachers’ classroom 


practices are consistently 


communicated, 


understood by most staff 


and faculty, and 
monitored throughout 
the school year. 


Existing instructional 
schedules include 
uninterrupted blocks of 
schoolwide learning 
time. Content instruction 
and instructional support 
staff are coordinated or 
systematically organized 
and aligned across grade 
levels and content areas. 


Specific or precise 
expectations for high- 
quality instruction are 
communicated and 
understood by most 
staff, monitored by 
school leaders, and 
consistently 
implemented by most 
teachers. 


Instructional schedules 
are developed in 
collaboration with 
teachers and ensure that 
instructional support 
staff are coordinated and 
aligned across grade 
levels and content areas 
to provide students with 
differentiated access to 
high-quality core 
instruction. An effective 
process is in place for 
evaluating the schedule 
based on collected data 
related to the quality of 
instruction and student 
needs across grade levels 
and content areas. 
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Limited evidence 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


aaenareliays4 


Sustaining 


2.3. Identifying and 
Addressing 
Student 
Academic 


Needs 


2.4 Classroom 
Observation 
Data Use 


2.5 Student 
Assessment Data 
Use (for 
schoolwide 
decision-making) 
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No formal data collection 
process is in place for 
identifying individual 
students’ academic 
needs. Specific protocols 
for using data and 
identifying actions to 
address student 
academic needs are not 
in place. 


Instructional leaders 
rarely or never conduct 
class observations (such 
as learning 
walkthroughs). Evidence 
that specific and 
actionable feedback on 
the quality and 
effectiveness of 
instruction is being 
provided to individual 
teachers is limited or 
nonexistent. 


Building and teacher 
leaders use little or no 
student assessment data 
to make decisions 
related to schoolwide 
practices. 


Formal strategies and 
processes (such as 
instructional leadership 
team, collaborative 
planning, and 
professional learning 
communities) are in 
place, with protocols for 
using data and 
identifying actions to 
address individual 
students’ academic 
needs. However, the 
protocols may not be 
consistently used or 
followed. 


Instructional leaders 
conduct occasional or 
routine classroom 
observations (such as 
learning walkthroughs), 
primarily as a function of 
the principal’s role and 
with little to no timely 
feedback, focused on 
strengthening teachers’ 
instructional practices. 
Observation and 
feedback may be 
focused on only a few 
grades or subject areas. 


Building and teacher 
leaders consider only 
student results on state 
assessments when 
making decisions related 
to schoolwide practices. 


Formal strategies and 
processes (such as 
instructional leadership 
team, collaborative 
planning, and 
professional learning 
communities) and 
protocols for using data 
and identifying actions to 
address individual 
students’ academic 
needs are in place and 
used consistently, but 
communication among 
all staff about action 
steps is limited. 


Instructional leaders 
conduct regular 
classroom observations 
(such as learning 
walkthroughs) to gauge 
the quality of 
instructional practices 
and provide specific and 
actionable feedback on 
the quality and 
effectiveness of 
instruction. However, 
this information or these 
data do not inform 
instructional 
conversations or the 
provision of targeted and 
individualized supports 
(such as coaching) for 
teachers, as needed. 


Building and teacher 
leaders occasionally 
consider student results 
on benchmark and 
common assessments in 
addition to state 
assessments when 
making decisions related 
to schoolwide practices. 


Formal teaming and 
collaboration strategies, 
processes (such as 
instructional leadership 
team, collaborative 
planning, and 
professional learning 
communities), and 
protocols are 
consistently employed to 
address individual 
students’ academic 
needs by using data, 
identifying actions to 
address student learning 
needs, and regularly 
communicating action 
steps among all staff and 
teams to build and 
sustain a professional 
culture of learning. 


Instructional leaders 
conduct weekly or daily 
classroom observations 
(such as learning 
walkthroughs) focused 
on strengthening 
teachers’ instructional 
practices and provide 
specific and actionable 
feedback on the quality 
and effectiveness of 
instruction to individual 
teachers and teacher 
teams. These data 
inform instructional 
conversations and the 
provision of targeted and 
individualized supports 
(such as coaching) for 
teachers, as needed. 


Building and teacher 
leaders consistently use 
student results on 
benchmark and common 
assessments and state 
assessments to make 
decisions related to 
schoolwide practices. 
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Limited evidence 


Turnaround Practices and Indicators (TP&I) rating 


Developing 


agente liays4 


STU elialiayss 


2.6 Student 
Assessment Data 
Use (for 
classroom 
instruction) 


2.7 Structures for 
Instructional 
Improvement 


There is little or no 
emerging awareness of 
best practices for 
analyzing student 
performance data to 
inform instruction and 
assess progress toward 
intended student 
outcomes, or the effect 
of these practices is 
negligible. 


Structures, practices, 
and use of resources 
(such as collaborative 
meeting time, coaching, 
supports for 
implementing the 
Massachusetts 
Curriculum Frameworks) 
to support the use of 
assessment data and 
research-based 
instructional strategies 
and differentiation and 
to ensure rigor and 
relevance are limited, do 
not exist, or have 
negligible impact. 


Some teachers are aware 
of the importance of 
using a variety of 
assessment data to 
inform instruction and of 
employing research- 
based instructional 
strategies to discern 
progress toward 
intended student 
outcomes. However, not 
all staff consistently use 
these practices. 


Structures, practices, and 
use of resources (such as 
collaborative meeting 
time, coaching, supports 
for implementing the 
Massachusetts 
Curriculum Frameworks) 
to support the use of 
assessment data and 
research-based 
instructional strategies 
and differentiation to 
ensure rigor and 
relevance are in place 
but may be poorly 
defined, inefficient, or 
ineffective. 


Source: Massachusetts Department of Elementary and Secondary Education. 
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Most teachers are aware 
of their roles and 
responsibilities for using 
a variety of assessment 
data to inform 
instruction and for 
employing research- 
based instructional 
strategies to discern 
progress toward 
intended student 
outcomes. However, 
there are some barriers 
to using data effectively 
to improve instruction. 


Structures, practices, and 
use of resources (such as 
collaborative meeting 
time, coaching, supports 
for implementing the 
Massachusetts 
Curriculum Frameworks) 
to support the use of 
assessment data and to 
guide and select 
research-based 
instructional strategies 
and differentiation are 
clearly defined but are 
not always used 
consistently throughout 
the school. 


Most teachers work 
individually and 
collaboratively to use a 
variety of assessment 
data (such as common 
assessment data and 
student work) to discern 
progress toward 
intended student and 
school outcomes, 
determine appropriate 
action steps, and 
monitor the results of 
those actions. 


Structures, practices, 
and use of resources 
(such as collaborative 
meeting time, coaching, 
supports for 
implementing the 
Massachusetts 
Curriculum Frameworks) 
to support data-driven 
instruction and the use 
of research-based 
instructional strategies 
and differentiation are in 
place and consistently 
implemented, resulting 
in rigorous instruction— 
reflective of the shifts in 
cognitive demand for the 
Massachusetts 
Curriculum 
Frameworks—that 
meets the needs of each 
student. 
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Turnaround Practice Area 3. Student-Specific Supports and Instruction to All Students 


The school is able to provide student-specific supports and interventions informed by data and by the 
identification of student-specific needs (table B3). 


Tarel{erheele 


Limited evidence 


Table B3. Turnaround Practice Area 3. Student-Specific Supports and Instruction to All Students 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


Limited evidence 


Sustaining 


3.1 


3.2 


3.3 


General 
Academic 
Interventions 
and Enrichment 


Teacher Training 
to Identify 
Student Needs 
(academic and 
nonacademic) 


Determining 
Schoolwide 
Student 
Supports 
(academic 
interventions 
and enrichment) 
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Structured academic 
interventions and 
enrichment experiences 
(such as tiered system of 
support) are not in place 
or, if provided, are not 
based on research or 
promising practices. 


Staff members are 
provided with little or no 
training or support on 
how to identify and 
address student needs. 


Specific student 
academic intervention 
and enrichment needs 
are neither identified 
nor diagnosed. 


Specific, research-based 
interventions and 
enrichment experiences 


(such as tiered system of 
support) are defined and 


planned but might not 
be consistently or 
systematically 
implemented or 


available to all students. 


Some staff members are 


provided with training 
and support on how to 
identify and address at 
least one area of 
student need. However, 
training is not provided 
to all appropriate staff 
members or is not 
provided for all areas of 
student need (both 
academic and 
nonacademic). 


Specific student 
academic intervention 
and enrichment needs 
are diagnosed and 
identified annually or 
once a semester. 


Specific, research-based 
interventions and 
enrichment experiences 
(such as tiered system of 
support) are defined and 
planned and regularly 
provided, although 
student participation is 
not systematic, or 
interventions are not 
comprehensive (for 
example, available for 
both English language 
arts and math). Barriers 
might include scheduling 
conflicts or other 
structural challenges. 


Most staff members are 
provided with training 
and support to ensure 
that they can identify 
both academic and 
nonacademic student 
needs. However, staff do 
not receive training or 
support on how to 
respond appropriately 
to those cues, or staff 
fail to consistently 
respond to those cues 
despite training. 


Student academic 
performance is reviewed 
regularly throughout the 
school year to monitor 
progress and to identify 
emerging needs; 
however, students are 
not reassigned to 
interventions as needed 
throughout the school 
year. 


All students experience 
research-based 
academic interventions 
appropriate for their 
specific needs. These 
best practices and 
enrichment experiences 
are implemented 
systematically during 
regularly scheduled 
school time and for all 
core content areas 
through a robust tiered 
system of support. 


Most staff members are 
provided with training 
and support to ensure 
that they can identify 
cues when students 
need additional 
assistance (both 
academic and 
nonacademic) and can 
respond appropriately to 
those cues. 


Student learning and 
academic performance 
are regularly reviewed 
(at least once a month) 
throughout the school 
year, using a wide array 
of ongoing assessments 
to identify student- 
specific and schoolwide 
emerging needs. 
Students are reassigned 
to interventions, 
enrichment, and 
supports, as needed, 
throughout the school 
year. 
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3.4 


3.5° 


Multitiered 
System of 
Support 
(academic and 
nonacademic) 


Academic 
Interventions for 
English Language 
Learners 
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Limited evidence 


No system is in place to 
guide how to identify 
students in need of 
support or the necessary 
interventions and 
supports for those 
students. Leaders have 
not defined intervention 
entry and exit criteria 
for struggling students. 
Students are assigned to 
interventions using a 
wide range of 
information and 
processes that are not 
consistent across the 
school. 


Specific, research-based 
interventions for English 
learner students are not 
in place or, if provided, 
are not based on 
research or promising 
practices. 


Turnaround Practices and Indicators (TP&l) rating 


Developing 


Leaders have defined 
but not clearly 
communicated 
intervention entry and 
exit criteria for identified 
struggling students. 
Students are assigned to 
interventions with a 
limited application of 
the entry criteria, and 
student progress is not 
consistently or 
systemically monitored 
during the school year. 
The system meets one 
of the following three 
conditions: staff 
members follow 
consistent rules and 
procedures to identify 
students in need of 
additional assistance; a 
team of appropriate 
staff and stakeholders 
makes decisions about 
needed interventions 
and supports; or staff 
members follow 
consistent rules and 
procedures when 
monitoring the delivery 
and effectiveness of 
interventions and 
supports. 


Specific, research-based 
interventions for English 
learner students are 
defined and planned but 
might not be 
consistently or 
systematically 
implemented (due to 
staffing, scheduling, or 
other barriers) or 
designed to meet 
students’ specific needs. 


Limited evidence 


Leaders and teachers 
understand and use 
systems with criteria 
and protocols for 
identifying students for 
interventions and 
enrichment. Students 
are assigned to 
interventions, but this 
system meets only two 
of the following three 
conditions: staff 
members follow 
consistent rules and 
procedures to identify 
students in need of 
additional assistance; a 
team of appropriate 
staff and stakeholders 
makes decisions about 
needed interventions 
and supports; or staff 
members follow 
consistent rules and 
procedures when 
monitoring the delivery 
and effectiveness of 
interventions and 
supports. 


Specific, research-based 
interventions for English 
learner students are 
defined and planned and 
regularly provided. 
However, student 
participation is not 
always systematic, and 
supports are not always 
aligned to students’ 
specific needs. 


Sustaining 


Leaders and teachers 
actively use established 
systems with criteria 
and protocols for 
identifying students for 
interventions and 
enrichment. This system 
meets all of the 
following conditions: 
staff members follow 
consistent rules and 
procedures to identify 
students in need of 
additional assistance; a 
team of appropriate 
staff and stakeholders 
makes decisions about 
needed interventions 
and supports; and staff 
members follow 
consistent rules and 
procedures when 
monitoring the delivery 
and effectiveness of 
interventions and 
supports. 


All English learner 
students experience 
research-based 
academic interventions 
that are appropriate for 
their specific needs and 
that are implemented 
systematically in the 
school. 
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Turnaround Practices and Indicators (TP&l) rating 


axelKersixelg Limited evidence 1BY=\ 10) 0) fay = Mia avik=em=\YAlel=1arer=) Sustaining 
3.6" Academic Specific, research-based Specific, research-based Specific, research-based All students with 
Interventions for interventions for interventions for interventions for disabilities receive 
Students with students with disabilities students with disabilities students with disabilities research-based 
Disabilities are not in place or, if are defined and planned are defined and planned academic interventions 
provided, are not based but might not be and regularly provided. appropriate for their 
on research or consistently or However, student specific needs. These 
promising practices. systematically participation is not supports are 


implemented (because always systematic, and implemented 

of staffing, scheduling, | supports are not always systematically in the 
or other barriers) or aligned to students’ school. 

designed to meet specific needs. 

students’ specific needs. 


a. High Standards (formerly indicator 3.5 in the 2014/15 monitoring site visit report) was removed and content was incorporated into indicator 2.7 for the 
2015/16 monitoring site visit report. Academic Interventions for English Language Learners is now reflected in indicator 3.5. 

b. Academic Interventions for Students with Disabilities (indicator 3.6) was a new indicator for the 2015/16 monitoring site visit report. 

Source: Massachusetts Department of Elementary and Secondary Education. 


Turnaround Practice Area 4. School Climate and Culture 


The school provides a safe, orderly, and respectful environment for students and a collegial, collaborative, and 
professional culture among teachers (table B4). 


Table B4. Turnaround Practice Area 4. School Climate and Culture 


Turnaround Practices and Indicators (TP&I) rating 


axe lfersixelg Limited evidence 1BY=\-1e} o) faye Limited evidence Sustaining 
4.1 Schoolwide No schoolwide behavior The schoolwide behavior The schoolwide behavior The schoolwide 
Behavior Plan plan guides the plan includes adefined planincludesadefined behavior plan includes a 
consistent set of behavioral set of behavioral defined set of behavioral 
implementation of expectations, but there expectations, anda expectations, and the 
behavior management  isnotaclearsystemor system and set of system and set of 
procedures, orifthereis set of structures for structures for positive structures for positive 
a behavior plan, itis not positive behavioral behavioral supports are behavioral supports are 
implemented supports that is aligned aligned with those aligned with those 
consistently. with those expectations. expectations. However, expectations. In 
In addition, there is there is either no addition, most staff 
limited evidence that evidence that any staff implement the 
any staff implement the implement the procedures outlined in 


procedures outlined in procedures outlinedin — the schoolwide behavior 
the schoolwide behavior the schoolwide behavior plan. Leaders monitor 


plan. plan or evidence that implementation using 
only some staff data. 
implement the 
procedures. 
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Tatelerheele 


Limited evidence 


Turnaround Practices and Indicators (TP&I) rating 


1BY=\=1e) o) fay = 


MTaatinctem=alel=aras) 


Sustaining 


4.2 Adult-Student 
Relationships 


4.3 Expanded 
Learning 


4.4 Wraparound 
Services and 
External Partners 
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Structures (such as 
structured advisories 
and mentor programs) 
to support the 
development of strong, 
supportive relationships 
between adults and 
students are not in place 
or are inadequate. 


Students have limited to 
no opportunities to 
participate in expanded 
learning programs. 


There is little or no 
emerging leadership and 
no staff awareness of 
strategies to increase 
the capacity of families 
to support education in 
the home through 
wraparound services 
(such as health care and 
housing referrals). 


Structures (such as 
structured advisories 
and mentor programs) 
to support the 
development of strong 
relationships are defined 
but might not be used 
consistently or might not 
be available to all 
students. 


Opportunities for 
students to participate 
in expanded learning 
programs exist but might 
not be well defined, or 
awareness of and 
participation in the 
programs might be 
limited. 

Leaders and staff are 
aware of the needs of 
families to support 
education through 
wraparound services 
(such as health care and 
housing referrals). 
However, there is no 
system to provide these 
services consistently. 


Structures (such as 
structured advisories 
and mentor programs) 
are in place to support 
relationships among 
students and adults and 
to deliver social- 
emotional supports. 


Structured opportunities 
for students to 
participate in expanded 
learning programs are in 
place and are well 
defined. 


Leaders and staff are 
aware of the needs of 
families to support 
education through 
wraparound services 
(such as health care and 
housing referrals) and to 
provide these resources 
to families, as needed. 


Structures (such as 
structured advisories 
and mentor programs) 
are in place to support 
relationships among 
students and adults and 
to deliver social- 
emotional supports. 
These supports are 
monitored actively to 
discern whether they 
are meeting the needs 
of the school. 

All students have access 
to expanded learning 
opportunities that are 
well defined and well 
supported. High-need 
students are targeted 
for participation in these 
programs. 


Leaders and staff share 
individual and mutual 
responsibility for 
building the capacity of 
families to support 
education through a 
systemic system of 
wraparound services 
(such as health care and 
housing referrals). 
Leaders and staff assess 
the needs of students 
and families throughout 
the school year. 
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Tatelerheela 


Limited evidence 


Turnaround Practices and Indicators (TP&l) rating 


1BY=\=1e) o) faye 


Limited evidence 


Sustaining 


4.5° Family and There is little or no The school makes family The school makes family The school makes family 
Community evidence that the school and community engagement a priority, | and community 
Engagement makes family and engagement a priority, but only three or four of engagement a priority 


community engagement 
a priority. 


but only one or two of 
five conditions are met: 
one or more staff 
members coordinate 
family and community 
engagement activities; 
regular social events are 
planned throughout the 
year to engage families 
and community 
members; regular 
activities are planned 
throughout the year to 
engage families and 
community members in 
planning for and 
collaborating in the 
implementation of 
academic and 
nonacademic supports; 
staff members routinely 
reach out to families to 
communicate 
information about their 
children’s progress and 
needs; and 
communications with 
families are made 
available in multiple 
languages, as needed. 


five conditions are met: 
one or more staff 
members coordinate 
family and community 
engagement activities; 
regular social events are 
planned throughout the 
year to engage families 
and community 
members; regular 
activities are planned 
throughout the year to 
engage families and 
community members in 
planning for and 
collaborating in the 
implementation of 
academic and 
nonacademic supports; 
staff members routinely 
reach out to families to 
communicate 
information about their 
children’s progress and 
needs; and 
communications with 
families are made 
available in multiple 
languages, as needed. 


and all of the following 
five conditions are met: 
one or more staff 
members coordinate 
family and community 
engagement activities; 
regular social events are 
planned throughout the 
year to engage families 
and community 
members; regular 
activities are planned 
throughout the year to 
engage families and 
community members in 
planning for and 
collaborating in the 
implementation of 
academic and 
nonacademic supports; 
staff members routinely 
reach out to families to 
communicate 
information about their 
children’s progress and 
needs; and 
communications with 
families are made 
available in multiple 
languages, as needed. 


a. Formerly indicator 4.6 in the 2014/15 monitoring site visit report. 
Source: Massachusetts Department of Elementary and Secondary Education. 
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Table BS summarizes average ratings and standard deviations by practice area for the first three years among 43 
schools with three or more years of data. 


Table B5. Average Turnaround Practice and Indicators ratings and standard deviations by practice area for the 
first three years among 43 schools with three or more years of data, 2014/15-2018/19 


W/Kek-Vaachalaveas (cie-lalefelaemel=\Urevarela)) 
el md faatemo) mantelalinolalayss 


WUUldat-lael0latom olg-(oid(o-W-]a-t-m-laremlalel(er-ine)g First year Second year Third year 


1. Leadership, Shared Responsibility, and Professional Collaboration 


1.1 Use of Autonomy 1.95 (0.79) 2.49 (0.59) 2.72 (0.45) 
1.2 High Expectations and Positive Regard 1.98 (0.86) 2.26 (0.76) 2.47 (0.67) 
1.3 Vision/Theory of Action and Buy-In 1.65 (0.90) 2.14 (0.80) 2.44 (0.70) 
1.4 Monitoring of Implementation and School Progress 1.64 (0.79) 2.12 (0.79) 2.58 (0.66) 
1.5 Trusting Relationships 1.98 (0.94) 2.30 (0.77) 2.44 (0.73) 
1.6 Time Use for Professional Development and Collaboration 1.74 (0.69) 2.00 (0.62) 2.23 (0.65) 
1.7 Communication with Staff 1.79 (0.83) 2.14 (0.80) 2.35 (0.69) 
1.8 Sustainability 0.95 (0.51) 1.70 (0.74) 2.02 (0.83) 
2. Intentional Practices for Improving Instruction 

2.1 Instructional Expectations 1.77 (0.68) 1.81 (0.50) 1.98 (0.15) 
2.2 Instructional Schedule 1.58 (0.79) 1.84 (0.69) 2.28 (0.67) 
2.3 Identifying and Addressing Student Academic Needs 1.79 (0.86) 2.05 (0.72) 2.42 (0.70) 
2.4 Classroom Observation Data Use 1.95 (0.92) 2.30 (0.80) 2.70 (0.60) 
2.5 Student Assessment Data Use (for schoolwide decision-making) 1.98 (0.74) 2.51 (0.70) 2.60 (0.58) 
2.6 Student Assessment Data Use (for classroom instruction) 2.00 (0.79) 2.28 (0.80) 2.58 (0.54) 
2.7 Structures for Instructional Improvement 1.77 (0.68) 1.93 (0.51) 2.02 (0.15) 
3. Student-Specific Supports and Instruction to All Students 

3.1 General Academic Interventions and Enrichment 1.49 (0.63) 1.67 (0.61) 2.00 (0.76) 
3.2 Teacher Training to Identify Student Needs (academic and 1.07 (0.67) 1.63 (0.72) 1.91 (0.84) 
nonacademic) 

3.3 Determining Schoolwide Student Supports (academic 1.81 (0.85) 2.26 (0.82) 2.19 (0.85) 
interventions and enrichment) 

3.4 Multi-tiered System of Support (academic and nonacademic) 1.26 (0.79) 1.74 (0.85) 1.81 (0.85) 
3.5 Academic Interventions for English Language Learners 1.55 (0.76) 1.81 (0.66) 2.07 (0.74) 
3.6 Academic Interventions for Students with Disabilities 1.45 (0.60) 1.67 (0.64) 1.98 (0.64) 
4. School Climate and Culture 

4.1 Schoolwide Behavior Plan 1.60 (0.93) 2.02 (0.67) 2.21 (0.64) 
4.2 Adult—Student Relationships 1.33 (0.71) 1.74 (0.49) 1.98 (0.51) 
4.3 Expanded Learning 1.67 (0.81) 1.93 (0.88) 2.19 (0.96) 
4.4 Wraparound Services and External Partners 1.49 (0.70) 1.93 (0.59) 1.98 (0.71) 
4.5 Family and Community Engagement 1.95 (0.72) 2.28 (0.59) 2.37 (0.54) 


Note. Analysis is based on observation data from 43 schools with three or more years of data. The average Turnaround Practices and Indicators ratings are 
calculated on a four-point scale ranging from 0 (limited evidence) to 3 (sustaining). 
Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 
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Appendix C. The hierarchical linear models and detailed regression results 


The study team used hierarchical linear models (HLM) to investigate the relationships between Turnaround 
Practices and Indicators (TP&l) ratings and school mean student growth percentiles (SGP) in English language arts 
and math and chronic absenteeism rate.’ The analyses included all low-performing schools with at least one year 
of indicator data. In estimating the relationship between each indicator and outcome, the HLM analyses controlled 
for school-level covariates, including grade span (elementary or middle/high), percentages of male and female 
students, percentage of racial/ethnic minority students, percentage of English learner students, percentage of 
students with a disability, and percentage of economically disadvantaged students (as defined by the 
Massachusetts Department of Elementary and Secondary Education).? 


For each turnaround practice indicator the study team used the following two-level HLM model to estimate the 
relationships between an indicator rating and a given outcome after controlling for school covariates. As specified 
in the model, each school had one to five observations at level 1 (L1, the year level), depending on how many 
years of indicator data were available for the school. The L1 outcome was the outcome in year / for school j, and 
the L1 equation included four year dummy variables in addition to the year-specific measures of school 
characteristics and ratings for each turnaround indicator. At level 2 (L2, school level) the intercept was modeled 
as a school-level grand mean at year 1 plus a random error, and all the L1 slopes were fixed to their grand means 
with no error terms at L2. The general equation for the full regression model is as follows: 


Level 1 (year level): 


Yij = Boj + fj Indicator;; + Yh2;School;; + 63 ;Year ij + Ei; 


where Jj; is the outcome (school mean SGP in English language arts, school mean SGP in math, or chronic 
absenteeism rate) in year i for school j; Boj is the intercept in school j, which can be interpreted as the first-year 
outcome for school j, adjusted for the TP&I rating and school characteristics; Bij is the relationship between the 
indicator rating and the outcome for school j; Ba; is a vector of the relationships between the school 
characteristics and the school outcome in year i for school j; B3; is a vector of the relationship between year 
dummy variables and the outcome for school j; Indicator; is the indicator rating in year i for school j; Schooljis a 
vector of grand-mean centered, school-level covariates in year i for school j; Year; is a vector of dummy variables 
indicating the nth year for which the school had observation data in year i for school j; and jis the year-level 
residuals. 


' The study team considered conducting a factor analysis to determine whether the indicators held together in each practice area, but the 
Massachusetts Department of Elementary and Secondary Education staff indicated that the results would be difficult to interpret and that 
indicator-level findings were preferred to aggregate-level findings in each practice area. Another reason for not creating higher-level 
constructs based on individual indicators is that the TP&lI ratings were not designed to reflect higher-level constructs. 

2 The indicator ratings might be related to these school demographic characteristics because of differences in how schools prioritize and 
implement various turnaround practices. For example, a school with a very high percentage of economically disadvantaged students might 
need to implement turnaround practices differently from a school with a lower percentage of such students. Without controlling for these 
factors, a model would allow their influence on the outcome to be implicitly reflected in the estimated relationships between the TP&l 
ratings and the outcome. Comparing findings from models with and without these covariates would facilitate understanding of these 
relationships. 
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Level 2 (school level): 


Ba; = Y10 
Boj = Y20 
Bs; = Y30 


where Yoo is the intercept, interpreted as the overall average across schools in their first year with available 
data, Yi9 is the average relationship between the indicator rating and the given outcome, yz is a vector of 
average relationships between the school-level covariates and the given outcome, Y3q is the average relationship 
between year dummy variables and the given outcome, and Ug; is the school-level residual. 


The study team estimated the model with one turnaround indicator included in the model at a time and compiled 
a list of coefficients related to the turnaround indicators within each turnaround practice area. (See select HLM 
estimates in tables C1—C3; indicators with significant estimates are shown in bold.) A potential limitation of 
estimating a single indicator per model is the increased risk for Type 1 errors resulting from multiple comparisons. 
Therefore, the study team applied the Benjamini—Hochberg correction for multiple comparisons to obtain an 
unbiased statistical significance for each indicator within a practice area, as recommended by the What Works 
Clearinghouse (What Works Clearinghouse, 2020). Because the Massachusetts Department of Elementary and 
Secondary Education wanted to identify which indicators were related to the outcomes, the study team flagged 
the indicators that have a statistically significant coefficient y,9. Indicators with an estimated regression 
coefficient that was larger than .25 of a standard deviation but that was not statistically significant were not 
highlighted because the result was likely due to chance. Taken together, the flagged indicators can help the state 
understand the extent to which the turnaround indicators are related to improving the outcome. 
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Table C1. Regression results for the relationship between each schoolwide turnaround indicator on the 
Turnaround Practices and Indicators (TP&lI) rubric and selected school demographic characteristics and school 
mean student growth percentile in English language arts, 2014/15-2018/19 


Percentage of economically 


Turnaround indicator Grade span? disadvantaged students 
Sitlatel-lae| S\tlarel-Iae| S\tlatel-lae} 
Tate [fer-hwelg Estimate error p-value Estimate lagelg p-value Estimate lagele p-value 
Covariates only ni ni ni 2.65 1.38 0.05 —0.15 0.06 0.02 
1.1 1.51 0.79 0.06 2.55 1.37 0.06 —0.16 0.06 0.01 
1.2 1.03 0.74 0.16 2.41 1.38 0.08 -0.15 0.06 0.02 
1.3 1.51 0.68 0.03 2.08 1.39 0.13 —0.16 0.06 0.01 
1.4 1.65 0.71 0.02 2.50 1.36 0.07 —0.17 0.06 0.01 
1.5 1.53 0.69 0.03 2.01 1.39 0.15 —0.15 0.06 0.01 
1.6 1.93 0.79 0.01 2.03 1.38 0.14 -0.15 0.06 0.01 
1.7 1.48 0.69 0.03 2.42 1.37 0.08 -0.17 0.06 0.01 
1.8 1.41 0.70 0.04 0.81 1.36 0.55 -0.15 0.06 0.02 
2.1 1.27 1.14 0.27 2.60 1.37 0.06 -0.15 0.06 0.02 
2.2 2.12 0.74 <0.01* 2.16 1.36 0.11 0.15 0.06 0.02 
2.3 1.43 0.70 0.04* 2.07 1.39 0.14 —0.16 0.06 0.01 
2.4 2.03 0.68 <0.01* 2.47 1.35 0.07 0.17 0.06 0.01 
2.5 2.04 0.77 0.01* 2.12 1.37 0.12 -0.15 0.06 0.01 
2.6 2.07 0.76 0.01* 2.09 1.37 0.13 —0.16 0.06 0.01 
2.7 2.62 1.07 0.01* 2.52 1.36 0.06 -0.15 0.06 0.02 
3.1 2.40 0.80 <0.01* 2.05 1.36 0.13 —0.15 0.06 0.02 
3.2 2.64 0.69 <0.01* 2.02 1.34 0.13 —-0.17 0.06 0.01 
3.3 1.64 0.68 0.02* 1.64 1.42 0.25 —0.16 0.06 0.01 
3.4 1.75 0.66 0.01* 2.20 1.37 0.11 —0.17 0.06 0.01 
3.5 2.47 0.75 <0.01* 0.87 1.32 0.51 -0.15 0.06 0.02 
3.6 2.29 0.82 0.01* 0.73 1.34 0.59 —0.16 0.06 0.01 
4.1 2.34 0.74 <0.01* 1.84 1.37 0.18 —-0.17 0.06 0.01 
4.2 1.99 0.92 0.03 2.51 1.36 0.07 —-0.14 0.06 0.02 
4.3 1.20 0.65 0.06 2.69 1.37 0.05 —0.14 0.06 0.03 
4.4 1.28 0.79 0.11 2.36 1.38 0.09 —0.16 0.06 0.01 
4.5 1.18 0.94 0.21 2.18 1.42 0.13 -0.15 0.06 0.02 


* Statistically significant at the p < .05 level after multiple comparison adjustment; p-values are unadjusted. 

ni is not included in the model. 

Note: Bold type identifies TP&I indicators with statistically significant estimates. Each row presents hierarchical linear model estimates for one turnaround 
practice indicator. Each two-level hierarchical linear model that has observations grouped within schools controlled for school demographic characteristics. 
For the multiple comparisons method the Benjamini—Hochberg correction was used to adjust statistically significant thresholds (p-values) within each 
domain. Analysis was based on 229 year-specific observations from 91 schools from 2014/15 through 2018/19. Average school mean student growth 
percentile in English language arts was 45.03, with a standard deviation of 8.29. 

a. Elementary grades versus middle or high school grades. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 
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Table C2. Regression results for the relationship between each schoolwide turnaround indicator on the 
Turnaround Practices and Indicators (TP&I) rubric and selected school demographic characteristics and school 
mean student growth percentile in math, 2014/15-2018/19 


Turnaround indicator Grade span? Percentage disadvantage 
S\tlatel-lae| Syit-larel-lae| Stlatel-lae| 
arelKersinelg Estimate lagele p-value Estimate lagele p-value Estimate lacey e p-value 
Covariates only ni ni ni 3.97 1.66 0.02 —0.12 0.08 0.11 
1.1 1.98 0.97 0.04 3.84 1.65 0.02 —0.14 0.08 0.08 
1.2 1.01 0.90 0.26 3.74 1.67 0.03 —0.13 0.08 0.09 
1.3 1.64 0.83 0.05 3.35 1.68 0.05 —-0.14 0.08 0.08 
1.4 1.90 0.86 0.03 3.84 1.65 0.02 -0.14 0.08 0.06 
1.5 2.22 0.83 0.01 3.05 1.67 0.07 -0.13 0.08 0.08 
1.6 2.30 0.96 0.02 3.22 1.67 0.05 —0.13 0.08 0.10 
1.7 1.68 0.83 0.04 3.70 1.65 0.03 -0.15 0.08 0.06 
1.8 0.55 0.85 0.52 3.16 1.66 0.06 —0.07 0.08 0.34 
2.1 3.45 1.37 0.01* 3.83 1.64 0.02 —0.13 0.08 0.10 
2.2 1.30 0.92 0.15 3.66 1.67 0.03 0.12 0.08 0.11 
2.3 1.42 0.85 0.09 3.39 1.69 0.04 —0.13 0.08 0.09 
2.4 1.65 0.83 0.05 3.80 1.65 0.02 —0.14 0.08 0.07 
2.5 1.90 0.94 0.04 3.48 1.67 0.04 —0.13 0.08 0.09 
2.6 3.42 0.91 <0.01* 3.06 1.63 0.06 —0.14 0.08 0.06 
2.7 3.84 1.29 <0.01* 3.77 1.63 0.02 0.12 0.08 0.10 
3.1 2.34 0.98 0.02* 3.38 1.66 0.04 0.12 0.08 0.12 
3.2 2.19 0.86 0.01* 3.45 1.65 0.04 —0.14 0.08 0.06 
3.3 1.73 0.83 0.04* 2.91 1.72 0.09 -0.13 0.08 0.08 
3.4 1.95 0.81 0.02* 3.50 1.65 0.03 -0.15 0.08 0.06 
3.5 2.06 0.93 0.03* 3.05 1.63 0.06 —0.07 0.08 0.37 
3.6 1.46 1.01 0.15 3.02 1.65 0.07 —0.08 0.08 0.31 
4.1 1.46 0.92 0.11 3.47 1.69 0.04 0.14 0.08 0.08 
4.2 2.94 1.10 0.01* 3.76 1.64 0.02 0.12 0.08 0.13 
4.3 1.06 0.78 0.18 4.01 1.66 0.02 0.12 0.08 0.13 
4.4 1.41 0.95 0.14 3.65 1.67 0.03 0.14 0.08 0.07 
45 1.88 1.13 0.10 3.22 1.71 0.06 —0.13 0.08 0.10 


* Statistically significant at the p < .05 level after multiple comparison adjustment. p-values are the unadjusted values. 

ni is not included in the model. 

Note: Bold type identifies TP&I indicators with statistically significant estimates. Each row presents hierarchical linear model estimates for one turnaround 
practice indicator. Each two-level hierarchical linear model that has observations grouped within schools controlled for school demographic characteristics. 
For the multiple comparisons method the Benjamini—Hochberg correction was used to adjust statistically significant thresholds (p-values) within each 
domain. Analysis was based on 229 year-specific observations from 91 schools from 2014/15 through 2018/19. Average school mean student growth 
percentile in math was 43.62, with a standard deviation of 9.78. 

a. Elementary grades versus middle or high school grades. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 
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Table C3. Regression results for the relationship between each schoolwide turnaround indicator on the 
Turnaround Practices and Indicators (TP&lI) rubric and selected school demographic characteristics and school- 
level chromic absenteeism rates, 2014/15-2018/19 


Turnaround indicator Grade span? Percentage disadvantage 
Syt-larel-lae| SJtlarel-lae| Stlarel-lae| 
Nareltersixelg Estimate error p-value Estimate error p-value Estimate error p-value 
Covariates only ni ni ni -6.43 1.19 <0.01 0.16 0.05 <.01 
1 -1.91 1.00 0.06 -7.51 1.78 <0.01 0.13 0.08 0.11 
1.2 —3.29 0.93 <0.01* -6.81 1.75 <0.01 0.14 0.08 0.10 
1.3 -1.24 0.90 0.17 -7.04 1.82 <0.01 0.13 0.08 0.12 
1.4 —1.69 0.94 0.07 -7.31 1.78 <0.01 0.14 0.08 0.10 
1.5 -3.00 0.87 <0.01* -6.27 1.78 <0.01 0.13 0.08 0.12 
1.6 -1.91 1.03 0.07 —6.83 1.82 <0.01 0.12 0.08 0.15 
1.7 2.29 0.91 <0.01* -7.17 1.77 <0.01 0.15 0.08 0.07 
1.8 -0.70 1.00 0.49 —8.67 1.97 <0.01 0.12 0.09 0.18 
2.1 —1.16 1.46 0.43 -7.47 1.79 <0.01 0.12 0.09 0.15 
2.2 -1.54 0.99 0.12 —7.13 1.80 <0.01 0.12 0.08 0.15 
2.3 —2.08 0.90 0.02 -6.74 1.80 <0.01 0.13 0.08 0.13 
2.4 -0.29 0.91 0.75 —7.52 1.79 <0.01 0.13 0.09 0.14 
2.5 -1.75 1.02 0.09 -7.04 1.80 <0.01 0.12 0.08 0.14 
2.6 —-1.76 1.01 0.08 -7.07 1.80 <0.01 0.13 0.08 0.13 
2.7 -1.89 1.40 0.18 —7.43 1.78 <0.01 0.12 0.08 0.15 
3.1 —3.34 1.04 <0.01* —6.59 1.77 <0.01 0.11 0.08 0.19 
3.2 0.58 0.95 0.54 -7.70 1.80 <0.01 0.12 0.09 0.16 
3.3 —2.98 0.85 <0.01* —5.69 1.82 <0.01 0.14 0.08 0.08 
3.4 —1.63 0.88 0.06 -7.19 1.79 <0.01 0.15 0.09 0.09 
3.5 0.28 1.12 0.80 8.88 1.96 <0.01 0.13 0.09 0.17 
3.6 —2.49 1.16 0.03 —8.31 1.95 <0.01 0.13 0.09 0.15 
4.1 —3.20 0.94 <0.01* —6.53 1.77 <0.01 0.15 0.08 0.08 
4.2 —3.36 1.15 <0.01* -7.27 1.76 <0.01 0.11 0.08 0.17 
4.3 0.45 0.86 0.60 -7.55 1.79 <0.01 0.13 0.09 0.13 
4.4 -1.13 1.01 0.27 —7.35 1.79 <0.01 0.14 0.09 0.11 
4.5 —1.36 1.20 0.26 —6.97 1.86 <0.01 0.13 0.08 0.13 


* Statistically significant at the p < .05 level after multiple comparison adjustment. p-values are the unadjusted values. 

ni is not included in the model. 

Note: Bold type identifies TP&I indicators with statistically significant estimates. Each row presents hierarchical linear model estimates for one turnaround 
practice indicator. Each two-level hierarchical linear model that has observations grouped within schools controlled for school demographic characteristics. 
For the multiple comparisons method the Benjamini—Hochberg correction was used to adjust statistically significant thresholds (p-values) within each 
domain. Analysis was based on 229 year-specific observations from 91 schools from 2014/15 through 2018/19. Average school chronic absenteeism rate 
was 27.46, with a standard deviation of 12.22. 

a. Elementary grades versus middle or high school grades. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 
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Appendix D. Sensitivity analyses 


The study team chose to run a series of single-indicator models controlling for school-level covariates because the 
results would be easier to interpret for action by the Massachusetts Department of Elementary and Secondary 
Education and by low-performing schools (see tables 4—7 in the main report). To provide a more robust set of 
analyses that take into account multiple indicators, with and without school-level covariates, the study team 
conducted sensitivity analyses with models accounting for covariances among Turnaround Practices and 
Indicators (TP&lI) ratings in the same practice area in the rubric and examined the relationships between TP&l 
ratings and school outcomes unconditioned on school-level covariates. 


To investigate how all the indicators in a practice area together related to each outcome, all indicators in the same 
practice area were included in a linear regression model with school-level covariates controlled for (model | in 
tables D1—D3). Then the study team removed school-level covariates and re-ran the model, which yielded 
unconditional relationships between the TP&lI ratings and the outcomes (model II in tables D1—D3). Finally, to 
select a subset of indicators that are strongly related to school outcomes from model Il, the study team 
implemented a stepwise selection strategy to identify the subset of indicators that shared the most variance with 
the outcomes. Specifically, to enhance the accuracy of indicator selection, the least absolute shrinkage and 
selection operator (LASSO; Tibshirani, 1996) method was applied to cross-validate indicator selection (model III in 
tables D1—D3 presents selected indicators and the estimated regression coefficients). Findings from the sensitivity 
analyses serve as a robustness check of the main findings, and both sets of findings should be considered together. 


Compared with the single-indicator models reported in the main report, model! and model II yielded fewer 
statistically significant relationships for TP&l ratings and school mean student growth percentile (SGP) in English 
language arts and math and in some cases for chronic absenteeism rate (see tables D1—D3). Indicators in each 
practice area were expected to be related to each other because they measured related practices. However, 
because indicators in the same practice area were related to each other (see tables D4—D7), the shared variance 
among the indicators in a practice area reduced the unique explanatory power of any one indicator to the 
outcomes, which could result in a more conservative finding of a nonsignificant coefficient for that indicator. 


Moreover, including all TP&l indicators in a model would suggest that any regression coefficient for an indicator 
is “conditioned” on the other indicator ratings being held constant, which could decrease the interpretability of 
the findings. For example, the study team noticed some correlations in an undesired direction between some 
indicator ratings and chronic absenteeism rate. This could indicate that all schools were practicing on the same 
level across the indicators. Consider Turnaround Practice Area 2, for which a higher rating of TP&I indicator 2.4 
Classroom Observation Data Use was related to higher chronic absenteeism rates. Although all schools are unlikely 
to have the same practice level across all indicators, the results might also suggest that schools with higher chronic 
absenteeism rates were more likely to implement 2.4 Classroom Observation Data Use to a greater extent. 


Finally, the stepwise regression with a LASSO cross-validation approach (model Ill) extended model II and distilled 
a subset of indicators that shared the greatest variance with the outcomes. Those indicators represented a subset 
of the indicators highlighted in the main report that had similar magnitudes in the regression coefficients for the 
school mean SGPs in English language arts and math, which offered evidence of the validity of the main findings. 
For the chronic absenteeism outcomes, however, some findings were inconsistent. In particular, for three 
indicators, estimated coefficients were related to worse chronic absenteeism rates. 
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Table D1. Alternative regression coefficients for models of the relationship between turnaround indicators in 
the Turnaround Practices and Indicators (TP&I) rubric and school mean student growth percentile in English 
language arts, 2014/15-2018/19 


NV/KeYe (=) be 


Ill. Stepwise 


PN fave ferchne) as TPN Tate fercl xo) acy deletion with 
Tarelbret=YemnVidal included without the LASSO 
TP&I indicator covariates covariates maakeaaareye| 


1. Leadership, Shared Responsibility, and Professional Collaboration 


1.1 Use of Autonomy —0.66 —0.70 Removed 
1.2 High Expectations and Positive Regard 0.53 0.33 Removed 
1.3 Vision/Theory of Action and Buy-In 0.96 0.94 Removed 
1.4 Monitoring of Implementation and School Progress 0.44 —0.31 Removed 
1.5 Trusting Relationships -—0.37 —0.44 Removed 
1.6 Time Use for Professional Development and Collaboration 1.69 1.75 Removed 
1.7 Communication with Staff —0.41 —0.26 Removed 
1.8 Sustainability 0.28 0.98 Removed 


2. Intentional Practices for Improving Instruction 


2.1 Instructional Expectations —1.55 —1.62 Removed 
2.2 Instructional Schedule 1.57 1.79* Removed 
2.3 Identifying and Addressing Student Academic Needs —1.07 -0.31 Removed 
2.4 Classroom Observation Data Use 1.06 0.03 Removed 
2.5 Student Assessment Data Use (for schoolwide decision-making) 0.98 1.39 Removed 
2.6 Student Assessment Data Use (for classroom instruction) 1.26 1.26 Removed 
2.7 Structures for Instructional Improvement 0.92 0.64 Removed 


3. Student-Specific Supports and Instruction to All Students 


3.1 General Academic Interventions and Enrichment 0.60 0.97 Removed 
3.2 Teacher Training to Identify Student Needs (academic and 1.20 0.88 Removed 
nonacademic) 

3.3 Determining Schoolwide Student Supports (academic 0.68 0.15 Removed 
interventions and enrichment) 

3.4 Multi-tiered System of Support (academic and nonacademic) 0.08 —0.28 Removed 
3.5 Academic Interventions for English Language Learners 1.19 2.12* 3.01 
3.6 Academic Interventions for Students with Disabilities 0.37 0.10 Removed 


4. School Climate and Culture 


4.1 Schoolwide Behavior Plan 1.86* 1.56 Removed 
4.2 Adult—Student Relationships 0.76 —0.10 Removed 
4.3 Expanded Learning 0.59 1.22 Removed 
4.4 Wraparound Services and External Partners 0.17 0.17 Removed 
4.5 Family and Community Engagement —0.08 0.95 Removed 
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*Statistically significant at the p < .05 level. 

LASSO is least absolute shrinkage and selection operator. 

Note: Each row presents the regression coefficient for one indicator rating. The value of a regression coefficient can be interpreted as the difference in an 
outcome related to a one unit difference in the rating. Analysis based on 229 year-specific observations from 91 schools. 

a. With all indicators within each practice area included in the model, model | controlled for school demographic characteristics (grade span, percentages of 
male and female students, percentage of racial/ethnic minority students, percentage of English learner students, percentage of students with a disability, 
and percentage of economically disadvantaged students) and dummy variables for the years in which the schools had observation data. Model II did not 
account for school demographic characteristics. Model Ill applied a stepwise deletion method with the LASSO cross-validation approach to select a subset 
of items from model II. The statistical significance is not tested in model III. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D2. Alternative regression coefficients for models of the relationship between turnaround indicators in 
the Turnaround Practices and Indicators (TP&l) rubric and school mean student growth percentile in math, 
2014/15-2018/19 


NV/KeYe (=) be 


TRAN late lfersine) es Ill. Stepwise 


|. All indicators are terel=xe} deletion with 
Tae lrel=re nda) without the LASSO 
MUUladat-laelelalem ole-loid(a-w-la-r-W-laremiarel (reine) s covariates covariates raat=iaarele| 


1. Leadership, Shared Responsibility, and Professional Collaboration 


1.1 Use of Autonomy —0.33 —1.02 Removed 
1.2 High Expectations and Positive Regard —1.11 -0.53 Removed 
1.3 Vision/Theory of Action and Buy-In 1.03 1.57 Removed 
1.4 Monitoring of Implementation and School Progress 0.64 0.35 Removed 
1.5 Trusting Relationships 0.09 —0.43 Removed 
1.6 Time Use for Professional Development and Collaboration 2.11 2.71* Removed 
1.7 Communication with Staff 0.16 0.26 Removed 
1.8 Sustainability —0.69 0.12 Removed 


2. Intentional Practices for Improving Instruction 


2.1 Instructional Expectations 0.88 0.47 Removed 
2.2 Instructional Schedule 0.22 1.13 Removed 
2.3 Identifying and Addressing Student Academic Needs —1.63 —1.07 Removed 
2.4 Classroom Observation Data Use —0.18 —1.32 Removed 
2.5 Student Assessment Data Use (for schoolwide decision-making) 0.26 0.91 Removed 
2.6 Student Assessment Data Use (for classroom instruction) 3.31* 3.72* 2.75 

2.7 Structures for Instructional Improvement 2.14 1.86 Removed 


3. Student-Specific Supports and Instruction to All Students 


3.1 General Academic Interventions and Enrichment 0.43 1.01 Removed 
3.2 Teacher Training to Identify Student Needs (academic and 0.47 0.84 Removed 
nonacademic) 

3.3 Determining Schoolwide Student Supports (academic 0.31 0.63 Removed 
interventions and enrichment) 

3.4 Multi-tiered System of Support (academic and nonacademic) 0.25 —0.03 Removed 
3.5 Academic Interventions for English Language Learners 1.48 2.44* 2.17 
3.6 Academic Interventions for Students with Disabilities —0.16 —0.57 Removed 


4. School Climate and Culture 


4.1 Schoolwide Behavior Plan 0.19 —0.09 Removed 
4.2 Adult-Student Relationships 2.33 1.52 Removed 
4.3 Expanded Learning 0.34 0.74 Removed 
4.4 Wraparound Services and External Partners 0.37 0.75 Removed 
4.5 Family and Community Engagement 0.85 2.15 Removed 
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*Statistically significant at the p < .05 level. 

LASSO is least absolute shrinkage and selection operator. 

Note: Each row presents the regression coefficient for one indicator rating. The value of a regression coefficient can be interpreted as the difference in an 
outcome related to a one unit difference in the rating. Analysis based on 229 year-specific observations from 91 schools. 

a. With all indicators within each practice area included in the model, model | controlled for school demographic characteristics (grade span, percentages of 
male and female students, percentage of racial/ethnic minority students, percentage of English learner students, percentage of students with a disability, 
and percentage of economically disadvantaged students), and dummy variables for the years in which the schools had observation data. Model II did not 
account for school demographic characteristics. Model Ill applied a stepwise deletion method with the LASSO cross-validation approach to select a subset 
of items from model II. The statistical significance is not tested in model III. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D3. Alternative regression coefficients for models of the relationship between turnaround indicators in 
the Turnaround Practices and Indicators (TP&I) rubric and school chronic absenteeism rate, 2014/15—-2018/19 
Model? 


TANI late fersine) es lll. Stepwise 


|. All indicators Tare Lure l=re| deletion with 
included with without idal=w WANSSS1@) 
mUUlaar-laellarem ola-ladie-w-la-y-W-laremlarelte-inele covariates covariates EJ eyelgey-loia) 


1. Leadership, Shared Responsibility, and Professional Collaboration 


1.1 Use of Autonomy 0.29 —0.07 Removed 
1.2 High Expectations and Positive Regard -3.94* —5.14* -4.71 

1.3 Vision/Theory of Action and Buy-In 1.95 0.84 Removed 
1.4 Monitoring of Implementation and School Progress —0.43 0.05 Removed 
1.5 Trusting Relationships —2.51 —2.12 Removed 
1.6 Time Use for Professional Development and Collaboration —0.02 -0.85 Removed 
1.7 Communication with Staff 0.34 —0.38 Removed 
1.8 Sustainability 1.64 2.81 Removed 


2. Intentional Practices for Improving Instruction 


2.1 Instructional Expectations 0.31 0.27 Removed 
2.2 Instructional Schedule —1.03 —2.47 —2.40 
2.3 Identifying and Addressing Student Academic Needs -1.99 —-3.01* —2.73 
2.4 Classroom Observation Data Use 2.04 3.31* 3.51 
2.5 Student Assessment Data Use (for schoolwide decision-making) -0.77 —3.13* —2.93 
2.6 Student Assessment Data Use (for classroom instruction) -0.75 0.19 Removed 
2.7 Structures for Instructional Improvement —0.32 1.05 Removed 


3. Student-Specific Supports and Instruction to All Students 


3.1 General Academic Interventions and Enrichment —3.05* —4,93* -4.97 
3.2 Teacher Training to Identify Student Needs (academic and 3.41* 4.47* 4.37 
nonacademic) 

3.3 Determining Schoolwide Student Supports (academic -3.08* —4,93* -5.06 
interventions and enrichment) 

3.4 Multi-tiered System of Support (academic and nonacademic) -0.35 -0.50 Removed 
3.5 Academic Interventions for English Language Learners 3.26* 4.72* 4.59 
3.6 Academic Interventions for Students with Disabilities —2.39 —2.57 —2.61 


4. School Climate and Culture 


4.1 Schoolwide Behavior Plan —2.75* —4.06* —3.02 

4.2 Adult—Student Relationships —2.47 —1.64 Removed 
4.3 Expanded Learning 1.59 1.74 Removed 
4.4 Wraparound Services and External Partners —0.06 0.15 Removed 
4.5 Family and Community Engagement 0.28 —1.48 Removed 


*Statistically significant at the p < .05 level. 

LASSO is least absolute shrinkage and selection operator. 

Note: Each row presents the regression coefficient for one indicator rating. The value of a regression coefficient can be interpreted as the difference in an 
outcome related to a one unit difference in the rating. Analysis based on 229 year-specific observations from 91 schools. 
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a. With all indicators within each practice area included in the model, model | controlled for school demographic characteristics (grade span, percentages of 
male and female students, percentage of racial/ethnic minority students, percentage of English learner students, percentage of students with a disability, 
and percentage of economically disadvantaged students), and dummy variables for the years in which the schools had observation data. Model II did not 
account for school demographic characteristics. Model III applied a stepwise deletion method with the LASSO cross-validation approach to select a subset 
of items from model Il. The statistical significance is not tested in model Ill. 

Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D4. Correlations among Turnaround Practices and Indicators (TP&l) indicators in the Leadership, Shared 
Responsibility, and Professional Collaboration practice area, 2014/15—-2018/19 
TP&l indicator 


TP&I indicator 


1.1 Use of Autonomy 1.00 

1.2 High Expectations and Positive Regard 49 1.00 

1.3 Vision/Theory of Action and Buy-In 58 .66 1.00 

1.4 Monitoring of Implementation and School Progress 53 53 .68 1.00 

1.5 Trusting Relationships 50 61 72 50 1.00 

1.6 Time Use for Professional Development and Collaboration 44 51 59 48 56 1.00 

1.7 Communication with Staff 42 61 62 48 .63 52 1.00 

1.8 Sustainability 50 56 59 53 55 57 48 1.00 
Mean 2.33. 2.18 2.00 2.12 2.19 1.97 2.09 1.68 
Standard deviation 0.75 0.77 086 0.83 0.82 0.71 0.82 0.84 


Note: The correlations were calculated as Spearman’s rank correlation coefficients given the ordinal nature of the indicator ratings. Analysis based on 229 
year-specific observations from 91 schools. 
Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D5. Correlations among Turnaround Practices and Indicators (TP&l) indicators in the Intentional 
Practices for Improving Instruction practice area, 2014/15-2018/19 


mUUldat-laelelarem ole-ladle-w-la-r-W-laremlarelle-inels 


TP&l indicator Bp D3} 2.4 2S) 2.6 

2.1 Instructional Expectations 1.00 

2.2 Instructional Schedule 45 1.00 

2.3 Identifying and Addressing Student Academic Needs A5 57 1.00 

2.4 Classroom Observation Data Use 50 51 .63 1.00 

2.5 Student Assessment Data Use (for schoolwide decision-making) 35 45 61 52 1.00 

2.6 Student Assessment Data Use (for classroom instruction) 53 45 .68 57 .61 1.00 

2.7 Structures for Instructional Improvement 47 41 56 50 49 .57 1.00 
Mean 1.81 183 2.01 2.23 2.31 2.25 1.82 
Standard deviation 0.48 0.76 O84 082 0.77 0.74 0.51 


Note: The correlations were calculated as Spearman’s rank correlation coefficients given the ordinal nature of the indicator ratings. Analysis based on 229 
year-specific observations from 91 schools. 
Source: Authors’ analysis of data for 2014/15-2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D6. Correlations among Turnaround Practices and Indicators (TP&l) indicators in the Student-Specific 
Supports and Instruction to All Students practice area, 2014/15-2018/19 
BUUlaat-laelelarem ole-lad(e-w-la-r-W-lalemlarelie-iaels 


TP&l indicator a2 3.3 3.5) 
3.1 General Academic Interventions and Enrichment 1.00 
3.2 Teacher Training to Identify Student Needs (academic and nonacademic) .60 1.00 


3.3 Determining Schoolwide Student Supports (academic interventions and .64 54 1.00 
enrichment) 


3.4 Multitiered System of Support (academic and nonacademic) 53 51 57 1.00 

3.5 Academic Interventions for English Language Learners 50 43 44 51 1.00 

3.6 Academic Interventions for Students with Disabilities 54 41 46 43 53 1.00 
Mean 1.67 152 2.00 1.56 1.81 1.73 
Standard deviation 0.73 081 0.89 0.82 0.73 0.67 


Note: The correlations were calculated as Spearman’s rank correlation coefficients given the ordinal nature of the indicator ratings. Analysis based on 229 
year-specific observations from 91 schools. 
Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 


Table D7. Correlations among Turnaround Practices and Indicators (TP&lI) indicators in the School Climate and 
Culture practice area, 2014/15-2018/19 


OUUldat-laelelarem ole-ladle-w-la-r-W-laremlarelle-inelg 


TP&I indicator 


4.1 Schoolwide Behavior Plan 1.00 

4.2 Adult—Student Relationships 48 1.00 

4.3 Expanded Learning .25 33 1.00 

4.4 Wraparound Services and External Partners 42 31 36 1.00 

4.5 Family and Community Engagement 42 33 .26 42 1.00 
Mean 1.97 172 1.97 1.82 2.25 
Standard deviation 0.78 0.63 0.85 0.71 0.64 


Note: The correlations were calculated as Spearman’s rank correlation coefficients given the ordinal nature of the indicator ratings. Analysis based on 229 
year-specific observations from 91 schools. 
Source: Authors’ analysis of data for 2014/15—2018/19 from the Massachusetts Department of Elementary and Secondary Education. 
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